Managing content displayed on a touch screen enabled device using gestures

ABSTRACT

Approaches are provided for managing content displayed on a touch screen enabled device. An approach includes detecting a gesture on a surface of a computing device. The approach further includes collecting data about touch events generated from the gesture on the surface of the computing device. The approach further includes evaluating the collected data to determine whether the gesture is recognized for changing a layout or format of content displayed on the computing device. The approach further includes performing a service call to an application programming interface of a layout tool to have the change in the layout or format of the content executed when the gesture is recognized. The approach further includes displaying the content on the computing device with the change in the layout or format.

FIELD OF THE INVENTION

The present invention generally relates to content management, and moreparticularly, to systems and methods for managing content displayed on atouch screen enabled device using gestures.

BACKGROUND

The pervasiveness of touch screen enabled devices is ever increasing,including both mobile device and stationary device types. For example,users can interact with a traditional desktop computer, a tabletcomputer, a mobile phone, and so on to access a variety of functionalityfor work and personal uses. Additionally, the variety of functionalitythat is available to users of touch screen enabled devices alsocontinues to increase.

However, interaction by a user with touch screen enabled devices, andrelated display content such as text within documents, web pages,notifications, etc., can be hindered by the size and position of thedisplay content submitted to the display of the touch screen enableddevices. For example, the size and position of the display content canprovide inadequate viewability and readability of the display content tothe user, who typically views or reads the displayed content at anygiven moment. Consequently, having the appropriate layout or formattingof the display content is important at any given moment for any touchscreen enabled device type, usage, or user.

SUMMARY

In a first aspect of the invention, a method is provided for thatincludes detecting a gesture on a surface of a computing device. Themethod further includes collecting data about touch events generatedfrom the gesture on the surface of the computing device. The methodfurther includes evaluating the collected data to determine whether thegesture is recognized for changing a layout or format of contentdisplayed on the computing device. The method further includesperforming a service call to an application programming interface of alayout tool to have the change in the layout or format of the contentexecuted when the gesture is recognized. The method further includesreceiving the change in the layout or format of the content from thelayout tool. The method further includes displaying the content on thecomputing device with the change in the layout or format.

In another aspect of the invention, a computer program product isprovided for managing a display of content within an opened electronicdocument on a computing device. The computer program product includes acomputer readable storage medium having program instructions embodiedtherewith. The computer readable storage medium is not a transitorysignal per se, and the program instructions are readable by a computingdevice to cause the computing device to perform a method that includesperforming an initial service call to an application programminginterface of a layout tool to obtain functionality for recognizinggestures that are not presently recognized by an application that isbeing used to display the content. The method further includes detectinga gesture for changing a number of columns of the content displayed onthe computing device. The method further includes collecting data abouttouch events generated from the gesture on a surface of the computingdevice. The method further includes evaluating the collected data usingthe obtained functionality for recognizing gestures to determine whetherthe gesture is recognized for changing the number of columns of thecontent displayed on the computing device. The method further includesperforming a service call to the application programming interface ofthe layout tool to execute the changing the number of columns of thecontent when the gesture is recognized. The method further includesreceiving the change in the number of columns of the content from thelayout tool. The method further includes displaying the content on thecomputing device with the change in the number of columns.

In a further aspect of the invention, a system is provided for thatincludes a CPU, a computer readable memory and a computer readablestorage medium. The system further includes program instructions toreceive user preferences for a layout or format of content displayed ona computing device. The system further includes program instructions todisplay the content on a display of the computing device based on theuser preferences. The system further includes program instructions toperform a first service call to an application programming interface ofa layout tool to obtain functionality for recognizing gestures that arenot presently recognized by an application that is being used to displaythe content on the computing device. The obtained functionality includescriteria supported by the layout tool for changing a number of columnsof the content. The system further includes program instructions todetect a gesture for changing the number of columns of the contentdisplayed on the computing device. The system further includes programinstructions to collect data about touch events generated from thegesture on a surface of the computing device. The system furtherincludes program instructions to compare the collected data to thecriteria to determine whether the gesture matches the criteria supportedby the layout tool for changing the number of columns of the content.The system further includes program instructions to perform a secondservice call to the application programming interface of the layout toolto execute the changing the number of columns of the content when thegesture matches the criteria. The system further includes programinstructions to receive the change in the number of columns of thecontent from the layout tool. The system further includes programinstructions to display the content on the display of the computingdevice with the change in the number of columns. The programinstructions are stored on the computer readable storage medium forexecution by the CPU via the computer readable memory.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present invention is described in the detailed description whichfollows, in reference to the noted plurality of drawings by way ofnon-limiting examples of exemplary embodiments of the present invention.

FIG. 1 shows an illustrative environment for implementing the steps inaccordance with aspects of the invention.

FIG. 2 shows an exemplary flow in accordance with aspects of theinvention.

FIGS. 3-6 show visual representations of examples of operation inaccordance with aspects of the invention.

FIG. 7 shows an exemplary flow in accordance with aspects of theinvention.

DETAILED DESCRIPTION

The present invention generally relates to content management, and moreparticularly, to systems and methods for managing content displayed on atouch screen enabled device using gestures. More specifically,implementations of the invention provide systems and methods thatrecognize gestures on a surface of touch screen enabled devices suchthat a user can change a layout or format of the content (e.g., text)displayed on the touch screen enabled devices. In embodiments, thegesture may include using two or more point of contact pinch and stretchgestures or using one or more point of contact tap and select gestures.The change in layout or format can include rearranging the content froma single column layout into a two or more column layout or merging thecontent from a two or more column layout into a number of columns thatis less than the two or more column layout such that readability orviewability of the display content is enhanced.

More specifically, in accordance with aspects of the present invention,systems and methods are provided for detecting a gesture performed by auser to create one or more columns of content from a selection of thecontent. In this regard, the position of the content within the one ormore columns is created based on points of contact detected from theuser's finger(s) or other mechanism of manipulation contacting a surfaceof the touch screen enabled devices when performing the gesture. Inaccordance with other aspects of the present invention, systems andmethods are provided for detecting a gesture performed by a user tomerge two or more columns of content from a selection of the content. Inthis regard, the two or more columns of content is selected based onpoints of contact detected from the user's finger(s) or other mechanismof manipulation contacting a surface of the touch screen enabled deviceswhen performing the gesture.

In accordance with further aspects of the present invention, systems andmethods are provided for detecting a gesture performed by a user tochange a number of columns of content from a selection of the content.In this regard, the number of columns of content is selected based on anumber of points of contact detected from the user's finger(s) or othermechanism of manipulation contacting a surface of the touch screenenabled devices when performing the gesture. In accordance with yetfurther aspects of the present invention, systems and methods areprovided for detecting a gesture performed by a user to display agraphical user interface (GUI) representing a layout or format ofcontent. In this regard, a default layout or format can be selected bythe user or the layout or format of the content can be changed (e.g., anumber of columns used to display the content may be increased ordecreased) based on the gesture.

In accordance with additional aspects of the present invention, systemsand methods are provided for defining user preferences for the layoutand format of content in order to automatically change the display ofthe content based on display size and characteristics of the touchscreen enabled device. In this regard, as thresholds are surpassed, thelayout and format of the content can be changed automatically (e.g., anumber of columns of content can be increased or decreased). The presentinvention provides many advantages over conventional touch screenenabled devices, which cannot separate and merge columns of content toenhance readability or viewability of the content.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

FIG. 1 shows an illustrative environment 10 for managing the processesin accordance with the invention. To this extent, environment 10includes a server 12 or other computing system that can perform theprocesses described herein. In particular, server 12 includes acomputing device 14. The computing device 14 can be resident on anetwork infrastructure or computing device of a third party serviceprovider (any of which is generally represented in FIG. 1).

The computing device 14 also includes a processor 20 (e.g., CPU), memory22A, an I/O interface 24, and a bus 26. The memory 22A can include localmemory employed during actual execution of program code, bulk storage,and cache memories which provide temporary storage of at least someprogram code in order to reduce the number of times code must beretrieved from bulk storage during execution. In addition, the computingdevice includes random access memory (RAM), a read-only memory (ROM),and an operating system (O/S).

The computing device 14 is in communication with external I/Odevice/resource 28 and storage system 22B. For example, I/O device 28can comprise any device that enables an individual to interact withcomputing device 14 (e.g., graphic user interface) or any device thatenables computing device 14 to communicate with one or more othercomputing devices using any type of communications link. The externalI/O device/resource 28 may be for example, a handheld device, PDA,handset, keyboard etc.

In general, processor 20 executes computer program code (e.g., programcontrol 44), which can be stored in memory 22A and/or storage system22B. Moreover, in accordance with aspects of the invention, programcontrol 44 controls a layout tool 50 and applications 55, which performprocesses described herein. The layout tool 50 and applications 55 canbe implemented as one or more program code in program control 44 storedin memory 22A as separate or combined modules. Additionally, the layouttool 50 and applications 55 may be implemented as separate dedicatedprocessors or a single or several processors to provide the function ofthe layout tool 50 and applications 55. While executing the computerprogram code, the processor 20 can read and/or write data to/from memory22A, storage system 22B, and/or I/O interface 24. The program codeexecutes the processes of the invention. The bus 26 provides acommunications link between each of the components in computing device14.

In embodiments, the layout tool 50 may be implemented within the O/S ofthe computing device 14 (e.g., a touch screen enabled device) or as astandalone application or module within the computing device 14, andmade available (e.g., made available as a service) to applications 55 ofthe computing device 14 by way of an application programming interface(API) framework. The API framework defines a set of functionalities ofthe layout tool 50 that are independent of their respectiveimplementation, allowing both definition and implementation to varywithout compromising each other. The API framework is used to allow forthe integration of new functionality (e.g., a touch event listeningservice) into the applications 55 of the computing device 14, or toshare data between the applications 55 and the layout tool 50.

By way of example, the layout tool 50 may be configured to providefunctionality via the API framework for determining whether a gesture isrecognized for changing a layout or formatting of content displayed onthe computing device 14. In embodiments, upon recognition of thegesture, the layout tool 50 may be further configured to process aservice call from one or more of the applications 55 to obtainfunctionality supported by the recognized gesture. The processing theservice call may include: (i) activating subroutines attached to therecognized gesture based on gathered data regarding the gesture; (ii)performing functionality supported by the recognized gesture on content(e.g., content displayed by one or more of the applications 55) suchthat the content is modified with the manipulation (e.g., a layout orformat change); and (iii) return the results of the service call (e.g.,the content modification) to one or more of the applications 55 suchthat the content modification can be displayed on a display of thecomputing device 14.

The computing device 14 can comprise any general purpose computingarticle of manufacture capable of executing computer program codeinstalled thereon (e.g., a personal computer, a mobile device, a server,etc.). However, it is understood that computing device 14 is onlyrepresentative of various possible equivalent-computing devices that mayperform the processes described herein. To this extent, in embodiments,the functionality provided by computing device 14 can be implemented bya computing article of manufacture that includes any combination ofgeneral and/or specific purpose hardware and/or computer program code.In each embodiment, the program code and hardware can be created usingstandard programming and engineering techniques, respectively.

Similarly, server 12 is only illustrative of various types of computerinfrastructures for implementing the invention. For example, inembodiments, server 12 comprises two or more computing devices (e.g., aserver cluster) that communicate over any type of communications link,such as a network, a shared memory, or the like, to perform the processdescribed herein. Further, while performing the processes describedherein, one or more computing devices on server 12 can communicate withone or more other computing devices external to server 12 using any typeof communications link. The communications link can comprise anycombination of wired and/or wireless links; any combination of one ormore types of networks (e.g., the Internet, a wide area network, a localarea network, a virtual private network, etc.); and/or utilize anycombination of transmission techniques and protocols.

FIGS. 2 and 7 show exemplary flows for performing aspects of the presentinvention. The steps of FIGS. 2 and 7 may be implemented in theenvironment of FIG. 1, for example. As noted above, the flowchart(s)illustrate the architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program products asalready described herein in accordance with the various embodiments ofthe present invention. The flowchart and block diagrams in the Figuresillustrate the architecture, functionality, and operation of possibleimplementations of systems, methods and computer program productsaccording to various embodiments of the present invention. In thisregard, each block in the flowchart or block diagrams may represent amodule, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

FIG. 2 is a flow diagram illustrating a process 200 for managing thedisplay of content on a touch screen enabled device in accordance withaspects of the present invention. At step 205, a user performs a gesturein an attempt to manipulate or modify a layout or format of content thatis displayed on a display of a touch screen enabled device. A gesture isthe physical act or motion performed on, or by, the input device(finger, fingers, pen/stylus, mouse, and so on). For example, to launch,activate, or invoke a command, a user may use a single finger to tap thetouch screen enabled device (equivalent to a left-click with a mouse, atap with a pen, or Enter on a keyboard). The manipulation is the ongoingreaction or response (e.g., a layout or format change) that the object(e.g., the content) or user interface has to the gesture. For example,both a pinch and stretch gesture may cause content to shrink and expandin some manner (e.g., increase or decrease a number of columns of text).The final outcome of the manipulation, how the manipulation ismanifested by the content on the screen and in the user interface, isthe interaction. Interactions depend on how a manipulation isinterpreted and the command or action that results from themanipulation. For example, a number of columns of content can beincreased or decreased through a pinch or stretch gesture, but theresults may differ depending on a number of points of contact used inthe gesture (e.g., the number of columns of content created can beincreased by increasing a number of points of contact in the gesture).

The content may be text provided within one or more electronic documents(e.g., a Microsoft® Word document or an Adobe® portable document, whichare registered trademarks of Microsoft Corporation and Adobe SystemInc., respectively, in the United States and/or other countries) thatare opened using an appropriate document generating and/or editingapplication (e.g., Microsoft® Office or Adobe® Reader®, which areregistered trademarks of Microsoft Corporation and Adobe System Inc.,respectively, in the United States and/or other countries) on a touchscreen enabled device (e.g., computing device 14 as described withrespect to FIG. 1). In embodiments, the one or more electronic documentsmay be any type of document, such as a report, a letter, a news article,a book, a travel document, a time card, a purchasing or sales agreement,a drawing, an image, a spreadsheet, or other electronic form such as aweb page or web document (e.g., a something that has a uniform resourceidentifier (URI) and can return representations (responses in a formatsuch as HTML or JPEG or RDF) of the identified resource in response toHTTP requests), a file or self contained piece of information availableto a computing device, artwork, or a blueprint, etc. It should beunderstood by those of ordinary skill in the art that the presentinvention is not limited by the type of electronic document that isopened.

At step 210, the gesture on the touch screen enabled device is detectedand captured. For example, the appropriate document generating and/orediting application (e.g., one or more of the applications 55 asdescribed with respect to FIG. 1) may be configured to detect andcapture a user initiated gesture on the touch screen enabled device byinheriting touch interaction properties of the layout tool (e.g., layouttool 50 as described with respect to FIG. 1). More specifically, theapplication can be configured to leverage the API framework of thelayout tool in order to obtain additional functionality (e.g., add onservices such as layout or format modification) available from touchinteraction properties (e.g., recognized gestures) of the layout tool.In embodiments, the touch screen enabled device includes multi-touchtechnology, which provides for the ability of a surface (e.g., the touchscreen or trackpad) to recognize the presence of one or more points ofcontact with the surface. This one or more contact point awareness isused to implement the additional functionality to activate certainsubroutines attached to predefined gestures defined within the layouttool.

At step 215, the detection and capture of the gesture includes gatheringdata about the touch events on the touch screen enabled device. Thedetection of the gesture may start when a user first touches the touchscreen with a finger or other mechanism of manipulation (e.g., astylus), continues as the device tracks the position of the user'sfinger(s) or other mechanism of manipulation, and ends by capturing afinal event of the user's fingers or other mechanism leaving the screen.Throughout this interaction, positional data of the content (e.g.,paragraph and layout of an existing display format such as a singlecolumn of text), positional data of the user's finger(s) or othermechanism of manipulation delivered to the device (e.g., positional dataof points of contact on the surface of the touch screen device andchanging positional data of the points of contact indicative of motionby the points of contact), and the number of points of contact made bythe finger(s) or other mechanism of manipulation delivered to the deviceto make the gesture (e.g., two points of contact spreading apart over asingle column of text may denote splitting a single column of text intotwo columns of text) provides the details of every interaction.

At step 220, the gathered data about the touch events on the touchscreen enabled device is evaluated to determine whether the gathereddata matches any criteria as a recognized or known gesture. For example,the gathered data is evaluated to determine whether the gathered datamatches any criteria supported by the layout tool (e.g., layout tool 50as described with respect to FIG. 1) and/or the application (e.g., oneor more of the applications 55 as described with respect to FIG. 1) as arecognized gesture. In embodiments, the application may includefunctionality for recognizing gestures, and can use the positional dataand the number of points of contact made by the user's finger(s) orother mechanism of manipulation delivered to the device to determine ifa gesture matches any criteria supported by the layout tool and/or theapplication.

In embodiments, the criteria for gestures that are supported by thelayout tool as a manipulation of content may include any number or typesof gestures. For example, as shown in FIG. 3, criteria for a stretchinggesture 300 can be used to activate a subroutine that creates two ormore columns of content 305 from a single column of content 310. In thisregard, the two or more columns of content 305 are created based on aposition of the one or more points of contact 315 when performing thegesture. Additionally, as shown in FIG. 4, the criteria for a pinchinggesture 320 can be used to activate a subroutine that combines two ormore columns of content 325 into a lesser number of columns of content330 (e.g., 2 into 1 or 3 into 2). In this regard, the two or morecolumns of content 325 are selected based on the points of contact 335when performing the gesture.

In additional embodiments the criteria for recognized gestures that aresupported by the layout tool as a manipulation of content may furtherinclude interaction designations such as a number of points of contactmade by the user's finger(s) or other mechanism of manipulation on thesurface of the touch screen enabled device. For example, two points ofcontact in a stretching gesture could match criteria and interactiondesignations indicative of changing the number of columns of contentfrom one column to two columns, and three points of contact in astretching or pinching gesture could match criteria and interactiondesignations indicative of changing the number of columns of contentfrom one column to three columns. Alternatively, two points of contactin pinching gesture could match criteria and interaction designationsindicative of merging the two columns under the two points of contact,and three points of contact in a pinching gesture could match criteriaand interaction designations indicative of merging the three columnsunder the three points of contact. In embodiments in which the layout ofthe application and/or computing device support a larger layout (e.g.,landscape mode), the layout tool may support four and five points ofcontact gestures or other mechanisms of manipulation.

In other embodiments, the criteria for recognized gestures that aresupported by the layout tool as a manipulation of content may furtherinclude any number or types of gestures made in combination with a GUI.For example, as shown in FIG. 5, a single column of content can be splitinto two or more columns of content using a tapping gesture 340 todisplay a GUI 345 comprising a series of images 350 representative of alayout or format of content (e.g., two dots representative of twocolumns of text). The default series of the images 350 may be indicativeof a default layout or format (e.g., two columns of content). From thedefault series of the images 350, additional gestures (e.g., one or moreadditional tapping gestures) may be used in combination with the GUI toaccept the default layout or format, increase the selected number ofimages, as shown in GUI 355 (e.g., select more dots to create morecolumns of content), or decrease the selected number of images, as shownin GUI 360 (e.g., select less dots to create less columns of content).

Similarly a user can select any content for further layout or formatchanges via any number or types of gestures that match additionalcriteria supported by the layout tool and/or the application. Forexample, further layout or format changes may include any resizing ofcontent (e.g., adjusting a relative size of one or more pieces ofcontent within a display area) by way of a pinching or stretchinggesture, any rotation of content (e.g., rotating one or more pieces ofcontent relative to a display area) by way of a turning gesture, or anyselecting and moving of content (e.g., selecting and dragging content toanother position within a display area to change an alignment of thecontent) by way of a tap or press and hold, and a slide or swipegesture. However, the above-mentioned additional criteria are merelyillustrative of the present invention and it should be understood bythose of ordinary skill in the art that the present invention is notlimited by the additional criteria supported by the layout tool or theappropriate document generating and/or editing application.

In embodiments, when the gesture is not recognized or unknown eitherbecause the application is not leveraging the API framework of thelayout tool in order to obtain criteria supported by the layout tool asa recognized gesture, or because the gesture does not match any criteriasupported by the layout tool or the application as a recognized gesture,then the process moves to step 225. At step 225, a notification isprovided to the user that the gesture is not recognized and instructionsare provided on known gestures. For example, the layout tool, O/S of thecomputing device, or application will notify a user via a pop-up windowor other notification service (e.g., a scrolling error notification)that the gesture is not recognized. Additionally, the layout tool, O/Sof the computing device, or application will provide the user with alist of known gestures supported by the layout tool and the application.In embodiments, the list of known gestures may also include details onthe interaction of content by way of the known gestures (e.g., how themanipulation is manifested by the content on the screen and in the GUI).

In embodiments, when the gesture is recognized or known because thegesture matches the criteria supported by the layout tool and/or theapplication as a recognized manipulation of content, then the processmoves to step 230. At step 230, the gathered data about the touch eventson the touch screen enabled device is evaluated to determine whether thegathered data matches more than one criterion as a recognized gesture.For example, the gathered data is evaluated to determine whether thegathered data matches more than one criterion supported by the layouttool (e.g., layout tool 50 as described with respect to FIG. 1) and/orthe application as a recognized attempt at a gesture. In embodiments,the application may include functionality for recognizing gestures, andcan use the positional data and the number of points of contact made bythe user's finger(s) or other mechanism of manipulation delivered to thedevice to determine if a gesture matches more than one criterionsupported by the layout tool and/or the application

In embodiments, when the gesture matches more than one criterionsupported by the layout tool and/or the application, then the processmoves to step 235. At step 235, a notification is provided to the userthat the gesture matches more than one criterion supported by the layouttool and/or the application. For example, the layout tool, O/S of thecomputing device, or application will notify a user via a pop-up windowor other notification service (e.g., a scrolling error notification)that the gesture meets more than one criterion supported by the layouttool and/or the application. Additionally, the layout tool, O/S of thecomputing device, or application will request that the user select oneof the operations supported by the more than one criterion that meet thegesture.

At step 240, when either: (i) the gesture does not match more than onecriterion supported by the layout tool and/or the application; or (ii)the user selects one of the operations supported by the more than onecriterion that matches the gesture (as discussed with respect to step235), then a service call is sent for the operation supported by thegesture. In embodiments, the service call is sent by the application andreceived by the layout tool. The service call includes the gathered dataabout the touch events (as discussed with respect to step 215) and canbe sent using any scheme (e.g., any calling convention) configured tosend the gathered data as parameters for the subroutines attached to thegesture that are defined within the layout tool.

At step 245, the subroutines attached to the gesture are activated basedon the gathered data, and the operation supported by the gesture isperformed on the content such that the content is modified (e.g., alayout or format change). In embodiments, the layout tool (e.g., layouttool 50 as described with respect to FIG. 1) activates the subroutinesto perform the operation and modify the content. The modification of thecontent is performed as a result of the interaction manifested by way ofthe detected manipulation of the content. For example, as shown in FIG.6, in the instance that text content 365 is changed from a single columnformat 370 to a two column format 375 (i.e., an interaction manifestedby way of a detected manipulation of the text content 365), the layouttool (e.g., layout tool 50 as described with respect to FIG. 1) may beconfigured to adjust the text content 365 (i.e., one or more of thecontents within the one or more electronic documents) in such a mannerthat the single column of text 370 is split into two columns of text375.

In embodiments, the results (e.g., the manipulation of the content) ofthe service call are returned from the layout tool to the applicationsuch that the application may display the final outcome of themanipulation, or how the manipulation is manifested by the content onthe display and in the user interface as the interaction.Advantageously, the present invention is capable of leveraging the APIframework of the layout tool in order to obtain additional functionality(e.g., add on services such as layout or format modification) availablefrom touch interaction properties (e.g., recognized gestures) of thelayout tool.

FIG. 7 is a flow diagram illustrating a process 400 for interacting withcontent on a computing device in accordance with embodiments of thepresent invention. At step 405, user preferences are defined for thelayout and format of content displayed on one or more computing devices.For example, a user may use the O/S of one or more computing devices todefine the layout and format preferences for content displayed on eachof the one or more computing devices. In embodiments, the preferencesmay be defined based on display size and/or characteristics of the oneor more computing devices. For example, a user may generally prefer tohave content displayed in a two column layout when viewing the contenton a nineteen-inch display but may also prefer to have content displayedin a three column format when viewing the content from a specific textediting application on the nineteen-inch display. Alternatively, a usermay generally prefer to have content displayed in a single column layoutwhen viewing the content on a four-inch display such as on a mobilecomputing device. In this regard, as thresholds (e.g., a size of thedisplay) are surpassed, the layout and format of the content can bechanged automatically (e.g., a number of columns of content can beincreased or decreased based on a size of the display). In furtherembodiments, other thresholds are contemplated by the present invention.For example, a threshold can be based on “words per column” such that asthe font size is increased, there would be less words in a column andmore hyphenated words. A user's preference might be to have fewercolumns if the number of words across a column is too few.

At step 410, the computing device displays at least a portion of contenton a screen display based on the defined preferences. For example, if apreference is defined that requires textual content be displayed in atwo column layout when the display size is greater than six-inches, anda user is using a computing device with a display of ten-inches, thenthe computing device will display the portion of content on the screenin a two-column layout. The content may be text provided within one ormore electronic documents that is opened using an application (e.g., oneor more of the applications 55 discussed with respect to FIG. 1).Accordingly, the portion of the content may be displayed using apredefined layout or format based on the defined preferences.

At step 415, a first service call is made to obtain functionality forgestures that are not presently recognized by the application. Inembodiments, the first service call is made to the layout tool (e.g.,the layout tool 50 discussed with respect to FIG. 1) and includesloading an API of the layout tool and making a request of the API eitherthrough unauthorized or authorized access. The request can beimplemented via a GET method that retrieves all criteria supported bythe layout tool as a recognized gesture. In embodiments, the criteriafor gestures that are supported by the layout tool as a manipulation ofcontent may include any number or types of gestures.

At step 420, an input (e.g., two points of content made with a surfaceof the touch screen enabled device) may be detected on a surface of thecomputing device. In embodiments, the detection of the input may bebased on resistive technology, surface acoustic wave technology,capacitive technology, infrared technology, optical technology, etc asshould be known to those of ordinary skill in the art. At step 425, inresponse to detecting the input, the application may begin to gatherdata regarding the input. The gathering of the data starts when a userfirst touches the touch screen in step 420, continues as the devicetracks the position of the user's finger(s) or other mechanism ofmanipulation, and ends by capturing a final event of the user's fingersor other mechanism leaving the screen. Throughout this interaction,positional data of the content (e.g., paragraph and layout of anexisting display format such as a single column of text), positionaldata of the user's finger(s) or other mechanism of manipulationdelivered to the device (e.g., positional data of points of contact onthe surface of the touch screen device and changing positional data ofthe points of contact indicative of motion by the points of contact),and the number of points of contact made by the finger(s) or othermanipulation delivered to the device to make the gesture (e.g., twopoints of contact spreading apart over a single column of text maydenote splitting a single column of text into two columns of text)provides the details of every interaction.

At step 430, the gathered data about the touch events on the touchscreen enabled device is evaluated to determine whether the gathereddata matches any criteria as a recognized gesture. For example, thegathered data is evaluated to determine whether the gathered datamatches any criteria supported by the layout tool (as retrieved in step415) or the application as a recognized attempt at the manipulation ofthe content though a gesture. In embodiments, the evaluation of thegathered data may include determining whether the gesture matches one ormore criterion as a recognized gesture, as previously discussed indetail with respect to steps 220-235 of FIG. 2.

At step 435, a second service call is made to obtain functionalitysupported by the recognized gesture. In embodiments, the second servicecall is made to the layout tool (e.g., the layout tool 50 discussed withrespect to FIG. 1) and includes making a request of the API eitherthrough unauthorized or authorized access. The request can beimplemented via a MODIFY method that includes the gathered data as theparameters of the method. In embodiments, subroutines attached to therecognized gesture are activated based on the gathered data, and thefunctionality supported by the recognized gesture is performed on theportion of content such that the portion of the content is modified(e.g., a layout or format change). In embodiments, the results (e.g.,the modification of the layout or format of the content) of the secondservice call are returned from the layout tool to the application (e.g.,one or more of the applications 55 discussed with respect to FIG. 1)such that the application may display the final outcome of themanipulation, or how the manipulation is manifested by the portion ofcontent on the display and in the GUI as the interaction.Advantageously, the present invention is capable of leveraging the APIframework of the layout tool in order to obtain additional functionality(e.g., add on services such as layout or format modification) availablefrom touch interaction properties (e.g., recognized gestures) of thelayout tool.

As should be understood by those of ordinary skill in the art, theabove-mentioned examples including modifications of content, aspects ofcontent, and adjustments of content are merely illustrative of thepresent invention, and thus the present invention is not limited byparticular modifications of content, aspects of content, and adjustmentsof content thereof discussed herein. For example, the present inventionis not limited to converting a single column of text to two or morecolumns of text, and may likewise be applied for adjusting images,audio, animations, etc. without departing from the spirit and scope ofthe present invention.

In embodiments, the invention provides a method that performs theprocess of the invention on a subscription, advertising, and/or feebasis. That is, a service provider, such as a Solution Integrator, couldoffer to provide the data collection and content modificationfunctionality on a network. In this case, the service provider cancreate, maintain, support, etc., a computer infrastructure, such ascomputer system 12 (FIG. 1) that performs the processes of the inventionfor one or more consumers. In return, the service provider can receivepayment from the consumer(s) under a subscription and/or fee agreementand/or the service provider can receive payment from the sale ofadvertising content to one or more third parties.

In still another embodiment, the invention provides acomputer-implemented method for collecting data and modifying content ona network. In this case, a computer infrastructure, such as computersystem 12 (FIG. 1), can be provided and one or more systems forperforming the processes of the invention can be obtained (e.g.,created, purchased, used, modified, etc.) and deployed to the computerinfrastructure. To this extent, the deployment of a system can compriseone or more of: (1) installing program code on a computing device, suchas computer system 12 (as shown in FIG. 1), from a computer-readablemedium; (2) adding one or more computing devices to the computerinfrastructure; and (3) incorporating and/or modifying one or moreexisting systems of the computer infrastructure to enable the computerinfrastructure to perform the processes of the invention

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method implemented in a computer infrastructurehaving computer executable code tangibly embodied on a computer readablestorage medium having programming instructions operable to: receive userpreferences for a layout or format of content displayed on a computingdevice; display the content on a display of the computing device basedon the user preferences; perform a first service call to an applicationprogramming interface of a layout tool to obtain functionality forrecognizing gestures that are not recognized by an application that isbeing used to display the content on the computing device, wherein theobtained functionality includes criteria supported by the layout toolfor changing a number of columns of the content; detect a gesture forchanging the number of columns of the content on a surface of thecomputing device; generate a list of known gestures supported by thelayout tool that provides details on how manipulation is manifested bythe content and a graphical user interface displayed by the computingdevice when the gesture does not match the criteria; perform a secondservice call to the application programming interface of the layout toolto execute the changing the number of columns of the content when thegesture matches the criteria; receive a change in the number of columnsof the content from the layout tool; display the content on thecomputing device with the change in the number of columns of thecontent; and further change automatically a layout or format of thechange in the number of columns of the content in response to thresholdsof the computing device being surpassed.
 2. The method of claim 1,wherein the computing device is a touch screen enabled device.
 3. Themethod of claim 2, wherein the programming instructions are furtheroperable to: provide a notification to a user that the gesture isunknown when the gesture is not recognized; and provide instructions tothe user on known gestures when the gesture is not recognized.
 4. Themethod of claim 2, wherein the programming instructions are furtheroperable to: provide a notification to a user that the gesture meetsmore than one criterion when the gesture meets more than one criterion;provide a request to the user for selection of one layout or formattingchange to be performed from one of the more than one criterion; andreceive a selection of the layout or formatting change.
 5. The method ofclaim 1, wherein the content is a single column of text, the gesture isa spreading gesture, and the layout or format change includes splittingthe single column of text into two or more columns of text.
 6. Themethod of claim 5, wherein the two or more columns of text is dependentupon the number of points of contact made by finger(s) or othermechanism of manipulation to the surface of the computing device toperform the gesture.
 7. The method of claim 1, wherein the content istwo or more columns of text, the gesture is a pinching gesture, and thelayout or format change includes combining the two or more columns oftext into a number of columns of text that is less than the two or morecolumns of text.
 8. The method of claim 7, wherein the number of columnsof text is dependent upon the number of points of contact made byfinger(s) or other mechanism of manipulation to the surface of thecomputing device to perform the gesture.
 9. The method of claim 1,wherein the content is a single column of text, the gesture is one ormore tap gestures, the one or more tap gestures activates a display of agraphical user interface and interacts with the graphical userinterface, and the layout or format change includes splitting the singlecolumn of text into two or more columns of text.
 10. The method of claim9, wherein the two or more columns of text is dependent upon theinteraction of the one or more tap gestures with the graphical userinterface for selecting the two or more columns of text.
 11. The methodof claim 1, wherein the content is two or more columns of text, thegesture is one or more tap gestures, the one or more tap gesturesactivates a display of a graphical user interface and interacts with thegraphical user interface, and the layout or format change includescombining the two or more columns of text into a number of columns oftext that is less than the two or more columns of text.
 12. The methodof claim 11, wherein the number of columns of text is dependent upon theinteraction of the one or more tap gestures on the graphical userinterface for selecting the number of columns of text.
 13. The method ofclaim 1, wherein a service provider at least one of creates, maintains,deploys and supports the computer infrastructure.